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SECTION  1 


OVERVIEW 


CONCEPT 

The  Software  Engineering  Exercise  (SEE)  is  a  risk  reduction  measure  designed  to  be  used  during 
the  source  selection  process  as  part  of  the  technical  evaluation  of  offerors.  It  is  considered  to  be  an 
effective  discriminator  in  reducing  the  risks  normally  associated  vith  a  software  acquisition.  The  SEE 
encompasses  the  development  and  administration  of  an  exercise  (test  problem)  that  is  implemented  by 
an  offeror  in  a  restricted  time  period  (usually  less  than  a  month).  The  exercise  problem  typically 
addresses  one  or  more  software  risk  areas  and  is  designed  to  be  evaluated  quickly  in  order  to  minimize 
its  schedule  impact  The  SEE  normally  requires  the  offerors  to  submit  a  draft  Software  Development 
Plan  (SDP)  with  their  proposals,  because  the  ability  to  follow  the  SDP  is  a  major  concern  of  the  SEE. 

While  the  SEE  is  designed  to  assist  in  the  technical  evaluat  ion  during  a  source  selection,  it  may 
also  be  applied  after  contract  award;  for  example,  as  a  work  task  n  a  Concept  Definition  (CD)  phase 
contract  prior  to  Full-Scale  Development  (FSD),  or  as  an  early  F  5D  phase  task.  When  used  in  these 
ways,  the  SEE  provides  an  early  indication  of  a  contractor's  abili  y  to  implement  his  SDP  and  provides 
an  early  focus  on  potential  problem  areas. 

The  purpose  of  this  document  is  to  provide  guidance  in  the  implementation  of  a  SEE;  it  is  not  a 
standard  for  performing  one.  There  are  too  many  variables,  including  number  of  offerors,  size  of 
contract,  risk  areas,  and  type  of  contract  (CD,  FSD,  demonstration,  preproduction)  to  specify  only  one 
SEE  method.  The  guidance  contained  in  this  document  is  meant  to  cover  the  major  SEE  activities;  the 
acquisition  agent  can  select  from  and  modify  these  activities  to  tailor  a  SEE  for  a  particular  application. 

If  time  and  staff  are  limited,  trade-offs  may  be  made  to  reduce  the  effort  to  perform  a  SEE  (streamlining 
of  the  SEE  is  discussed  in  section  2.) 

The  decision  to  use  the  SEE  should  take  into  account  many  factors  such  as  the  number  and 
severity  of  software  risks  in  the  procurement,  the  cost  of  the  SEE  in  terms  of  program  staff  and 
schedule,  the  cost  to  bidders  (the  SEE  may  be  particularly  burden  some  to  small  companies  forcing  them 
out  of  the  competition),  the  availability  of  experienced  program  office  software  staff,  and  some  factors 
mentioned  above  such  as  size  of  procurement,  number  of  bidders,  and  type  of  contract.  This  document 
provides  insight  into  the  effort,  time,  and  skills  required  to  implement  a  SEE,  and  in  that  way  is  an  input 
to  the  SEE  decision  process. 

OBJECTIVES 

Source  Selection 

When  used  during  a  source  selection,  the  SEE  helps  the  Government  determine  if  offerors  have 
the  software  engineering  capability  to  implement  their  proposals.  It  provides  a  means  to  evaluate  each 
offeror’s  software  development  process  including  their  requirements  analysis  approach,  design 
methods,  and  facility  with  the  proposed  or  required  design  and  implementation  languages.  It  requires 
the  offerors  to  apply  their  proposed  SDP  and  the  prescribed  tools  and  techniques,  and  to  demonstrate 
their  ability  to  apply  modem  software  engineering  practices  to  a  pioblem.  The  problem  itself  usually 
focuses  on  software  risk  areas  of  the  development  so  that  offerors  may  illustrate  their  knowledge  and 
expertise  in  pertinent  technology  areas. 
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Once  the  source  selection  is  completed  and  an  offeror  selected,  the  exercise  results  may  be  used 
further  to  alert  the  Government  to  areas  requiring  special  attention  after  contract  award.  Typically,  it  is 
not  discovered  until  the  first  delivery  of  a  Software  Requirements  Specification  (SRS)  that  the 
contractor  does  not  have  the  same  expectations  for  product  quality  as  the  Government.  If  such 
inconsistencies  exist,  they  may  be  addressed  by  amendments  to  the  Statement  of  Work  (SOW)  requiring 
the  contractor  to  enhance  his  software  development  process;  these  might  include  changes  to  his  software 
development  standards,  documentation  standards,  training  requirements,  or  tools. 

It  is  recommended  that  the  SEE  process  include  a  dry  run  of  the  exercise  problem  by  the 
Government  prior  to  its  delivery  to  the  offerors.  This  activity  can  improve  the  quality  of  an  acquisition 
by  identifying  potential  acquisition  problems  that  can  be  addressed  by  clarifications  or  changes  to  the 
specification,  SOW,  schedule,  or  other  contractual  documents.  The  dry  run  has  also  been  found  to  be  a 
most  effective  (and  cost-effective)  way  of  training  the  Government  team  in  current  software  engineering 
methods  and  software  acquisition  practices  that  it  will  apply  not  only  during  the  evaluation  of  SEE 
products,  but  during  FSD.  It  is  therefore  important  that  the  team  that  conducts  the  dry  run  and  evaluates 
the  SEE  products  continues  on  the  program  after  contract  award. 

After  Contract  Award 

If  the  SEE  is  not  used  as  part  of  the  source  selection  process,  it  may  be  used  after  contract  award 
to  provide  early  visibility  into  potential  problems  the  contractor  may  have  in  implementing  his  SDP.  It 
establishes  an  early  understanding  between  the  contractor  and  the  Government  of  the  expected  quality  of 
delivered  products  including  the  SRSs  and  subsequent  design  documents.  Also,  the  exercise  problem 
can  accomplish  some  early  contract  task,  such  as  a  prototyping  exercise,  considered  necessary  for  the 
system's  development.  When  used  during  a  CD  phase,  the  results  may  be  used  during  the  subsequent 
FSD  phase  source  selection  if  no  new  offerors  are  introduced. 
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SECTION  2 
SEE  PREPARATION 


PLANNING 

This  section  describes  the  activities  necessary  to  conduct  a  SEE.  Schedule,  cost,  and  staffing 
requirements  must  be  carefully  estimated  and  planned  so  that  the  scope  of  the  SEE  docs  not  exceed  the 
project's  budgets.  The  schedule  and  cost  associated  with  SEE  taslcs  varies  significantly  depending 
upon  whether  the  SEE  is  used  in  a  source  selection  or  after  contract  award,  and  upon  the  effort 
necessary  to  develop  and  dry  run  the  SEE  problem.  Figure  1  is  an  example  of  a  nominal  SEE  schedule 
utilizing  two  to  three  staff.  However,  as  mentioned  in  the  previous  section,  there  are  many  procurement 
variables  (size,  risks,  etc.)  that  can  influence  the  scope  of  the  various  SEE  activities  and  the  SEE  should 
be  tailored  accordingly.  Trade-offs  that  may  reduce  the  effort  required  are  discussed  at  the  end  of  this 
section  under  Streamlining. 

Experience  has  shown  that  more  effort  may  be  required  to  conduct  a  SEE  than  is  anticipated, 
particularly  the  preparation  and  dry  run  of  the  SEE  problem.  There  are  tasks  associated  with  the  dry  run 
that  are  not  readily  apparent;  for  example,  in  addition  to  the  effort  required  to  prepare  the  problem 
products  during  the  dry  run  (SRSs,  Software  Design  Documents  (SDDs),  etc.),  it  may  be  necessary  to 
develop  a  draft  SDP  (the  SEE  is  used  to  verify  the  use  of  the  SDP).  It  may  also  be  necessary  to  acquire 
computer  tools  and  to  train  the  dry  run  team  (for  example,  in  requirements  analysis  techniques,  design 
methods,  and  Ada);  and  to  resolve  problems  with  contractual  documents  (specification,  SOW, 
Instructions  for  Proposal  Preparation  (IFPP))  that  are  brought  to  li  ght  by  the  dry  run.  Each  of  these 
potential  tasks  must  be  planned,  and  trained  personnel  and  computer  resources  made  available. 

The  following  tasks  outline  the  process  of  preparing  a  SEE  and  are  discussed  individually  in  the 
following  paragraphs.  Next  to  each  task  is  an  estimated  range  of  the  number  of  staff  months  (SM) 
required  for  each  task;  the  wide  variation  is  a  function  of  the  number  of  software  risk  areas,  size  of 
procurement,  type  of  contract,  streamlining  effected,  and  other  procurement  variables. 

•  Preparing  the  Draft  Problem  Specification  (1-5  SM) 

•  Dry  Run  of  the  Problem  (2-10  SM) 

•  Preparing  the  Input  to  RFP  Documents  (1-3  SM) 

-  IFPP 

-  Evaluation  Criteria 

•  Preparing  the  SEE  Package  (2-5  SM) 

Detailed  Instructions  for  the  Offerors 

-  Final  Problem  Specification 


PREPARING  THE  PROBLEM  SPECIFICATION 

The  first  step  in  conducting  a  SEE  is  to  develop  the  exercise  problem.  The  problem  may  require 
the  offerors  to  analyze  performance  requirements;  illustrate  their  des  gn  method;  illustrate  their 
knowledge  of  one  or  more  high-risk  areas;  and  to  use  the  software  engineering  tools,  methods,  and 
techniques  contained  in  their  SDPs.  The  problem  must  be  relevant  to  the  software  risks  of  the  system 
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NOTE:  This  is  only  an  example!  Actual  schedules  may  vary  significantly  as  a  function  of  the 
tailoring  of  the  SEE  and  of  the  scope  of  the  exercise  problem. 


A  Prepare  Problem  A 

A  Dry  Run  of  Problem 

A 

A  Prepare  RFP  A 

Documents 

A  Develop  Evaluation  A 

A 

Criteria 

C 

A 

Prepare  SEE  A 

t 

1 

Package 

1 

V 

A  RFP  Release 

1 

t 

i 

e 

1  Proposal  A 

Preparation 

A  SEE  Release 

s 

A  Proposal  A 

Evaluation 

.  Contractor 

-  Government 

1  Perform  A 

Exercise 

A  Exercise  A 

Evaluation 

_ i _ i _ i _ i _ 

AOn-SiteA 
|  Audits 

0  12345  6789 

Months 

Figure  1 .  Example  SEE  Schedule 


being  acquired  and  restricted  enough  to  allow  requirements  analysis,  design,  and  possibly  some  coding 
to  be  accomplished  in  the  time  allocated  (usually  three  to  four  weeks). 

DRY  RUN  OF  THE  PROBLEM 

A  dry  run  of  the  exercise  problem  is  usually  conducted  to  develop  nominal  solutions  to  the 
problem,  to  discover  any  shortcomings  and  unintended  ambiguities,  to  develop  detailed  instructions  for 
the  offeror,  to  develop  problem  evaluation  criteria,  and  to  gain  knowledge  to  evaluate  the  SEE  products 
more  effectively.  Indirect  (but  perhaps  equally  important)  objectives  are  to  develop  team  skills  in 
requirements  analysis  methods,  design  methods,  design  languages,  the  implementation  language  (if 
specified),  and  applicable  standards  such  as  DOD-STD-2167A.  The  following  paragraphs  discuss  the 
major  tasks  associated  with  preparing  the  dry  run. 

Tools  and  Methods 

The  first  step  of  the  dry  run  is  to  select  the  tools  and  methods  to  be  used  and  to  become  familiar 
with  their  use.  These  might  include  required  standards  (DOD-STD-2167A,  MIL-STD-490,  and 
others),  requirements  analysis  methods,  design  methods  (structured,  object  oriented,  etc.),  design 
representation  techniques  (such  as  Buhr  diagrams),  a  program  design  language,  and  an  appropriate 
software  development  environment.  If  the  team  is  not  familiar  with  one  or  more  of  the  tools  and 
methods  to  be  employed,  appropriate  education  and  training  activities  must  be  arranged  (see  team 
training  below). 

Schedule 

The  example  schedule  in  figure  1  assumes  a  nominal  problem  development  effort  and  a  new  team 
with  limited  experience  in  the  problem  and  with  the  tools  and  methods  to  be  used.  The  time  required  to 
accomplish  the  dry  run  should  take  into  account  any  necessary  team  training  and  any  necessary 
refinement  and  clarification  of  the  problem  specification  resulting  from  the  dry  run. 

Team  Training 

If  the  dry  run  team  is  not  familiar  with  the  application,  the  requirements  analysis  and  design 
methods,  the  tools  to  be  used,  the  implementation  language  (if  specified),  or  a  Program  Design 
Language  (PDL),  then  time  must  be  allocated  for  training,  and  consultants  made  available  to  assist  as 
needed.  The  team  might  also  include  a  Government  representative  who,  among  other  things,  can  assist 
in  the  resolution  of  requirements  issues. 

Requirements  Analysis 

This  is  the  first  task  in  the  actual  dry  run  of  the  SEE  problem.  The  input  to  this  task  is  the  draft 
problem  specification.  The  products  of  this  task  typically  include:  a  software  architecture  that  identifies 
Computer  Software  Configuration  Items  (CSCIs),  the  technically  significant  sections  of  the  SRSs  for 
each  CSCI,  the  Interface  Requirements  Specifications,  and  data  and  control  flow  diagrams. 

Design 

This  dry  run  task  follows  requirements  analysis,  and  its  objective  is  to  raise  as  many  method  and 
design  issues  as  possible  to  prepare  for  the  offerors’  various  responses.  The  problem  should  require 
certain  difficult  areas,  (for  example,  interfaces,  timing,  control,  and  concurrency)  to  be  detailed  at  the 
Computer  Software  Component  (CSC)  level.  Some  coding,  unit  testing,  and  integration  could  also  be 
required  for  a  specified  function  or  set  of  functions. 
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Dry  Run  Analysis 

After  completing  the  dry  run,  an  evaluation  is  conducted  to  determine  if  clarifications  or  changes 
should  be  made  to  any  contractual  documents  (specification,  SOW,  schedule  and  so  on)  and  to  develop 
better  evaluation  criteria  for  each  factor  and  subfactor.  The  problem  specification  may  also  be  modified 
to  ensure  that  intended  requirements  analysis  and  design  trade-offs  are  addressed  by  the  offerors. 

PREPARING  THE  INPUT  TO  RFP  DOCUMENTS 

It  is  necessary  to  state  in  the  Request  for  Proposal  (RFP)  that  a  SEE  will  be  conducted  as  part  of 
the  source  selection  process  and  describe  what  it  entails.  It  is  also  necessary  to  include  the  criteria  to  be 
used  in  evaluating  an  offeror’s  response.  The  SEE  announcement  is  inserted  in  the  IFPP  section  of  the 
RFP  and  a  brief  discussion  of  what  the  SEE  entails  is  documented  in  the  Preliminary  Instructions  for 
the  Offerors  section.  The  detailed  instructions  for  the  offerors  and  the  final  problem  specification  are 
normally  delivered  to  the  offerors  upon  submission  of  their  proposals.  However,  if  the  SEE  is  released 
with  the  RFP,  then  it  would  contain  the  detailed  instructions  for  the  offerors  and  no  preliminary 
instructions  would  be  required.  It  is  also  necessary  to  state  in  the  RFP  that  a  draft  SDP  must  be 
submitted  with  the  proposal.  The  ability  of  the  offerors  to  follow  their  SDPs  is  a  major  factor  in  the 
SEE  evaluations. 

SEE  Announcement 

A  paragraph  similar  to  the  following  should  be  included  in  the  Instructions  For  Preparation  of 
Proposal  section  of  the  RFP.  It  states  that  a  SEE  will  be  conducted  and  refers  to  an  appendix  or 
attachment  that  contains  the  preliminary  instructions  for  the  offerors: 

"SOFTWARE  ENGINEERING  EXERCISE  (SEE) 

The  offeror  shall  be  given  a  problem  specification  defined  by  the  Government.  The  response 
will  be  evaluated  as  part  of  the  source  selection  process.  The  attached  Preliminary  Instructions 
for  the  Offerors  {appendix  A  of  this  document  contains  an  example)  describes  the  SEE  and 
the  SEE  products.  The  SEE  problem  specification  and  detailed  instructions  will  be  provided 
following  submission  of  proposals." 

Section  M,  Evaluation  Criteria 

A  statement  that  describes  the  general  criteria  to  be  used  to  evaluate  the  offerors'  SEE  products 
must  be  included  in  the  Evaluation  Criteria  section  of  the  RFP  (appendix  B  of  this  document  contains  an 
example).  The  evaluation  criteria  are  based  on  risk  areas  related  to  the  success  of  the  software 
development.  Lessons  learned  during  the  dry  run  can  be  instrumental  in  helping  to  establish  these 
evaluation  criteria.  Evaluation  standards  and  factors  are  developed  to  evaluate  both  the  SEE  products 
and  the  capabilities  of  the  offeror’s  staff.  These  are  discussed  in  section  3. 

PREPARING  THE  SEE  PACKAGE 

The  SEE  package  includes  two  documents:  1)  the  Detailed  Instructions  for  the  Offerors,  and  2) 
the  final  Problem  Specification.  It  is  recommended  that  the  package  be  delivered  to  the  offerors  upon 
submission  of  proposals  (this  is  done  so  that  the  offerors  may  concentrate  on  only  one  effort  --  the 
proposal  or  the  SEE  —  at  a  time).  This  also  allows  the  Government  to  concentrate  its  efforts  in  a  similar 
manner.  There  is  no  interaction  between  the  offerors  and  Government  during  the  offerors' 
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implementation  of  the  SEE,  allowing  the  Government  to  concentrate  on  the  technical  evaluation  of 
proposals  while  the  offerors  concentrate  on  the  SEE. 

Detailed  Instructions  for  the  Offerors 

Appendix  C  of  this  document  contains  an  example  of  the  Detailed  Instructions  for  the  Offerors. 
Typically  the  detailed  instructions  are  prepared  following  the  dry  run,  and  are  more  specific  and  detailed 
than  the  preliminary  instructions  included  in  the  RFP,  reflecting  changes  effected  as  a  result  of  the  dry 
run.  The  detailed  instructions  should  only  request  the  offerors  to  submit  products  that  demonstrate  the 
offeror's  competence  with  the  software  engineering  process,  and  provide  evidence  that  he  can  follow 
his  SDP;  the  list  of  requested  products  may  also  include  example  products  from  development  activities 
that  follow  detailed  design.  The  instructions  may  require  the  offeror  to  deliver  all  appropriate  products 
in  machine-readable  form  so  that  they  may  be  compiled  and  analyzed  as  part  of  the  evaluation.  Also, 
the  instructions  should  stipulate  that  all  participants  must  be  identified  in  the  proposal  as  members  of  the 
development  team  (no  consultants  or  ringers  are  allowed). 

Problem  Specification  (Final) 

The  problem  specification  is  typically  revised  as  a  result  of  the  dry  run  to  reflect  lessons  learned 
and  to  resolve  any  unintended  ambiguities  (appendix  D  contains  a  skeletal  example).  Some  ambiguities 
may  be  left  in  the  problem  to  measure  an  offeror’s  ability  to  detect  and  resolve  ambiguities  during 
requirements  analysis.  However,  it  is  particularly  important  to  remove  unintended  ambiguities  since  no 
communications  with  the  offerors  are  permitted  during  the  offerors'  performance  of  the  exercise 
problem. 

STREAMLINING 

Streamlining  is  a  desirable  goal  in  most  procurements,  but  something  must  be  given  up  to 
accomplish  the  streamlining  —  in  this  case,  some  of  the  risk  reduction  and  quality  of  results.  As  one 
person  said,  "The  best  SEE  streamlining  is  no  SEE."  This  is  true  if  the  streamlining  is  so  extreme  as  to 
jeopordize  the  integrity  of  the  SEE  results.  This  points  out  the  great  care  that  should  be  used  in 
eliminating  or  reducing  any  SEE  tasks  and  that  trade-offs  between  the  effort  saved  and  the  quality  of  the 
SEE  results  should  be  carefully  weighed.  Having  said  this,  the  following  paragraphs  present  some 
streamlining  ideas  both  for  source  selection  and  after  contract  award  applications  of  the  SEE. 

Source  Selection 

Streamlining  the  preparation  of  the  problem  specification  and  its  subsequent  dry  run  can  be 
accomplished  by  focusing  on  system  requirements  studied  during  the  preparation  of  the  system 
specification.  The  effort  to  prepare  and  dry  run  a  problem  can  be  reduced  if  it  is  analogous  to  one  that 
has  already  been  studied  ( for  example,  an  issue  from  the  system  specification  that  was  the  subject  of  a 
pre-RFP  study,  a  study  that  was  part  of  another  system  but  not  used  in  a  SEE,  or  a  classic  problem  for 
which  there  are  many  solutions).  Use  of  such  a  problem  reduces  the  time  to  develop  the  problem 
specification  as  well  as  the  time  to  perform  a  dry  run  since  solutions  would  already  have  been  studied. 
However,  care  must  be  taken  to  ensure  that  the  exercise  problem  is  unique  and  that  one  offeror  does  not 
have  any  more  advance  knowledge  of  the  problem  than  another. 

The  dry  run  of  the  exercise  problem  is  usually  the  most  time-consuming  task  of  conducting  a 
SEE.  A  SEE  may  be  implemented  with  a  minimal  dry  run,  but  this  should  be  considered  only  if  experts 
are  available  to  perform  the  evaluation  of  the  SEE  products  or  if  the  team  is  familiar  with  the  exercise 
problem;  for  example,  the  problem  was  the  subject  of  pre-RFP  studies.  The  SEE  has  been  given  to 
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contractors  during  a  CD  phase  without  a  dry  run  and,  used  in  this  way,  the  results  were  found  useful  in 
the  subsequent  FSD  source  selection. 

After  Contract  Award 

Some  of  the  SEE  tasks  may  be  tailored  out  or  accomplished  in  a  less  rigorous  and  structured 
manner  if  the  SEE  is  applied  after  contract  award,  since  the  results  do  not  affect  a  source  selection.  For 
example,  if  the  problem  is  extracted  from  the  system  segment  specification  and  was  also  the  subject  of 
pre-RFP  studies,  this  would  greatly  reduce  the  time  both  to  prepare  and  to  dry  run  the  SEE  problem. 
Other  streamlining  that  may  be  accomplished  in  an  after  contract  award  application  includes  replacing 
the  IFPP  documents  with  a  simplified  statement  of  criteria  and  with  a  simplified  set  of  instructions  to  the 
contractor.  The  evaluation  standards  and  factors  can  be  informal  since  the  results  are  not  for  source 
selection  use.  The  on-site  audit  can  be  a  technical  interchange  meeting  that  focuses  on  needed 
management  or  process  changes  rather  than  on  scoring  as  for  a  source  selection. 
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SECTION  3 


EVALUATING  THE  RESULTS 


FACTORS  AND  SUBFACTORS 

When  the  SEE  is  used  for  the  technical  evaluation  of  an  offeror  during  a  source  selection,  factors 
and  subfactors  must  be  developed  together  with  associated  evaluation  criteria  for  scoring  purposes. 
Some  of  the  example  evaluation  criteria  contained  in  appendix  B  could  be  broken  down  into  evaluation 
factors  and  subfactors  as  follows  (also  included  are  some  example  criteria): 

Factor  A,  Methods  Used 

Subfactor  1,  Requirements  Analysis  Methods 
Example  Criteria: 

•  Identification  and  resolution  of  ambiguities  in  the  system  specification 

•  Evidence  of  consistent  requirements  analysis  methods  and  tools 

•  Knowledge  of  method(s)  used 

•  Use  of  automated  tools 

Subfactor  2,  Design  Methods 
Example  Criteria: 

•  Evidence  of  consistent  design  methods  and  tools  across  the  system 

•  Support  for  control,  sequencing,  and  timing  functions,  not  just  data  flows 

•  Knowledge  of  method(s)  used 

Subfactor  3,  Transition  and  Traceability  Between  Requirements  Analysis  and  Design 
Methods 
Example  Criteria: 

•  Integration  of  methods  and  tools  to  allow  flow  from  requirements  to 
design  (and  back,  when  appropriate) 

•  Usefiil  mechanism  for  traceability  between  requirements  and  design 

Factor  B,  Requirements  Analysis 
Example  Criteria: 

•  Complete  set  of  software  requirements  and  derived  requirements,  but  not  design 

•  User  interfaces  clearly  defined 

•  All  assumptions  identified 

Factor  C,  Design 

Example  Criteria: 

•  Design  that  meets  requirements  without  introducing  limitations 

•  Clarity  of  design  and  understandable  text 

•  Recognition  and  treatment  of  exercise-specific  problem  areas  (these  can  be 
planned  into  the  problem,  and  specific  criteria  developed  for  each) 

Factor  D,  Team  Expertise 

Subfactor  1,  Knowledge  of  Methods 
Subfactor  2,  Team  Composition 
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STANDARDS 


After  factors  and  subfactors  have  been  identified,  standards  for  them  based  on  the  evaluation 
criteria  and  on  other  discriminators  discovered  during  the  dry  run  can  be  developed.  These  might 
include  such  things  as  the  identification  of  specification  ambiguities,  consistent  design  representation, 
performance  analysis,  and  behavior  analysis. 

EVALUATIONS 

Evaluations  are  normally  conducted  in  two  parts:  an  evaluation  of  the  submitted  SEE  products 
and  an  on-site  audit  It  is  important  that  the  same  team  that  evaluates  the  products  performs  the  on-site 
audit. 

Product  Evaluation 

The  products  are  evaluated  based  on  the  criteria  already  established,  resulting  in  an  initial  scoring 
for  each  offeror.  Questions  are  also  prepared  at  this  time  for  the  on-site  audit  to  verify  the  product 
evaluation  and  to  assess  the  offeror’s  SEE  team  capabilities  in  software  engineering,  Ada  (if 
appropriate),  selected  methods  and  tools,  and  so  forth. 

On-Site  Audit 

The  purpose  of  the  audit  is  to  verify  the  assessment  of  the  offerors'  products  and  to  assess  the 
capabilities  of  their  staffs.  The  offerors  brief  their  SEE  products,  methods,  tools,  and  other  factors. 

The  Government  then  has  an  opportunity  to  question  each  member  of  an  offeror's  team  to  evaluate  each 
team  member's  expertise.  The  offeror’s  briefing  cannot  address  anything  done  subsequent  to  the 
delivery  of  the  SEE  products. 

The  Government  has  the  opportunity  to  assess  the  organization's  use  of  software  development 
and  review  standards  and  procedures  as  well  as  management's  oversight  and  visibility  into  the 
development  of  the  SEE  products.  Experienced  staff  can  examine  areas  such  as  requirements 
derivation,  design  traceability,  and  communications  between  the  system  and  software  engineers.  The 
team  can  focus  on  the  offerors'  identification  of  software  development  risks  such  as  the  completeness, 
traceability,  and  testability  of  requirements;  knowledge  of  their  design/development  tools;  and  adherence 
to  their  SDP.  The  SEE  on-site  audit  should  be  able  to  answer  such  questions  as: 

•  Does  the  offeror’s  organization  practice  good  software  engineering  or  are  they  just  a  collection 
of  programmers? 

•  Does  the  offeror  know  how  to  use  the  proposed  tools? 

•  How  did  the  team  communicate  information  between  the  requirements  analysis,  design,  and 
implementation  phases? 

•  Was  a  mix  of  system  and  software  engineers  used  in  requirements  and  top-level  design  tasks? 

•  What  was  the  relationship  between  system  and  software  engineering? 

The  ability  for  the  Government  to  question  the  offerors  directly  has  been  very  revealing  and  was 
found  to  be  a  dominant  factor  in  the  evaluations.  If  streamlining  of  the  SEE  is  a  major  concern,  the  on¬ 
site  audit  may  result  in  the  most  cost-effective  utilization  of  resources. 
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SCORING 


The  SEE  scoring  approach  does  not  allow  the  clarification  request  and  deficiency  report  cycle 
associated  with  the  proposal;  the  offeror  is  allowed  only  one  submission  of  the  exercise  products  with 
no  changes  or  revisions.  However,  the  on-site  visit  provides  the  contractor  the  opportunity  to  clarify 
issues  for  the  Government.  The  scoring  of  the  SEE  may  be  pass/fail,  with  a  pass  if  the  offeror's 
products  are  at  least  satisfactory  for  a  prescribed  majority  (for  example,  65  or  75  percent)  of  the 
evaluation  factors.  A  better  and  more  precise  scoring  approach  can  be  taken  by  assessing  each 
evaluation  factor  for  strengths  and  weaknesses,  and  then  prioritizing  them  by  risk  to  give  weight  to  the 
more  important  risk  factors  for  the  procurement. 

CONTRACT  ADJUSTMENTS 

Various  problems  with  the  proposed  contractor’s  SEE  products  may  be  uncovered  during  the 
evaluation;  there  may  be  deficiencies  in  his  software  development  process  that  pose  a  threat  to  the 
success  of  the  acquisition.  Typically,  there  may  be  disparities  between  the  contractor's  SDP  and  his 
demonstrated  ability  to  implement  it.  These  deficiencies  may  be  addressed  by  negotiating  modifications 
to  the  SOW  that  require  the  contractor  to  perform  certain  tasks  to  improve  his  software  development 
process. 


t 


GLOSSARY 


Acronyms 


ADL  Ada-based  Design  Language 

CD  Concept  Definition 

CSC  Computer  Software  Component 

CSCI  Computer  Software  Configuration  Item 

FSD  Full-Scale  Development 

IFPP  Instructions  for  Proposal  Preparation 

PDL  Program  Design  Language 

RFP  Request  for  Proposal 

SDD  Software  Design  Document 

SDP  Software  Development  Plan 

SEE  Software  Engineering  Exercise 

SG  Scenario  Generator 

SOW  Statement  of  Work 

SRS  Software  Requirements  Specification 
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APPENDIX  A 


PRELIMINARY  INSTRUCTIONS  FOR  THE  OFFERORS  (Example) 


(This  is  only  an  example!  This,  document  should,  be  completely  tailored  for  each  SEE  as  the. 

instructions,  products,  gad, scope  am  differ  significantly  from  this,  example.) 

1.0  PURPOSE 

'  The  purpose  of  the  Software  Engineering  Exercise  (SEE)  is  to  permit  the  Government  to  evaluate 

an  actual  application  of  each  offeror’s  proposed  software  development  process  as  documented  in  the 
SDP.  The  SEE  concentrates  on  the  offeror’s  approach  to  requirements  analysis  and  design,  and  on 
•  their  interrelationship.  However,  the  offeror's  approach  to  implementation,  integration,  test,  quality 

assurance/configuration  management,  and  other  development  activities  explicitly  mentioned  in  the 
following  paragraphs  is  also  evaluated  by  the  Government  as  part  of  the  SEE.  (NOTE:  The  products  of 
the  SEE  can  be  expanded  to  include  limited  examples  of  products  from  these  activities.) 

2.0  INSTRUCTIONS 

The  offerors  will  provide  a  prototypical  example  of  their  proposed  software  development 
approach  as  applied  to  the  exercise  problem.  The  Government  will  define  the  problem  and  provide  the 
problem  specification  to  the  offerors  following  receipt  of  their  proposals.  In  performing  the  exercise, 
the  offerors  shall  comply  with  all  provisions  of  their  proposed  Software  Development  Plan;  deviations 
shall  be  noted  by  the  offerors. 

Participation  in  the  exercise  shall  be  limited  to  those  individuals  identified  in  the  offerer's 
proposal  as  part  of  the  full-scale  development  team.  Subcontractors  who  will  be  responsible  for 
1  software  development  shall  be  active  participants;  consultants  shall  be  precluded  from  participating. 

The  offerors  are  allocated  a  period  of  four  (4)  calendar  weeks  from  receipt  of  the  exercise 
materials  until  delivery  to  the  Government  of  all  requested  materials  in  the  formats  described  below. 

The  Government  then  evaluates  this  material.  Following  this  evaluation,  the  Government  conducts  on¬ 
site  visits  at  the  offerors'  facilities,  at  which  time  the  offerors  will  brief  the  Government  on  the  methods 
used  and  have  their  teams  available  to  answer  questions  from  the  Government.  The  Government  will 
coordinate  the  schedule  for  the  on-site  visits  with  the  offerors  upon  receipt  of  their  exercise  results. 

Note  that  there  will  be  no  interaction  between  the  offerors  and  the  Government  during  the  four-week 
exercise  period.  Should  the  offerors  have  any  questions  on  the  exercise,  they  are  instructed  to  identify 
appropriate  assumptions,  to  note  these  assumptions,  and  to  proceed  with  the  exercise  based  on  those 
assumptions. 

3.0  PRODUCTS  OF  THE  EXERCISE 

I  3.1  Delivery  to  the  Government 

At  the  conclusion  of  the  exercise,  the  offeror  shall  deliver  the  following  items  to  the  Government: 

!  a.  A  complete  software  architecture  for  the  sample  problem 

b.  For  one  or  more  Government-selected  components  of  the  system,  all  requirements  analysis 
conclusions  and  the  documentation 
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c.  For  one  or  more  Government-selected  components  of  the  system,  all  preliminary  design 
documentation,  including  requirements  traceability,  Ada-based  design  language  (ADL) 
listings,  and  graphics  products 

d .  For  one  or  more  Government-selected  components  of  the  system,  all  detailed  design 
documentation,  including  requirements  traceability,  ADL  listings,  and  graphics  products. 

All  textual  products  of  the  exercise,  including  requirements  analysis  conclusions  and 
documentation,  ADL  listings,  and  other  design  documentation  shall  be  delivered  to  the  Government 
both  in  hard  copy  form  and  in  machine-readable,  9-track,  1600/6250  bpi  tape  format  in  accordance  with 
ANSI  X3.27-1978  (exceptions  are  made  for  materials  which  the  offeror  does  not  propose  to  create  or 
maintain  on-line  during  the  contract).  In  particular,  graphical  representations  shall  be  submitted  in 
hardcopy  form;  the  offeror  shall  provide  six  (6)  copies  of  all  hardcopy  products.  The  products 
delivered  shall  be  clear,  coherent,  legible,  and  prepared  in  sufficient  detail  for  effective  evaluation; 
elaborate  documentation,  expensive  binding,  detailed  art  work,  or  other  embellishments  are 
unnecessary. 

3.2  Briefing  to  the  Government 

In  addition  to  the  products  described  above,  the  offerors  shall  provide  a  briefing  to  the 
Government  that  summarizes  their  experiences  in  the  carrying  out  of  the  exercise  and  describes  the 
products  produced.  The  briefings  shall  not  exceed  three  (3)  hours  in  duration.  The  topics  presented 
shall  include  the  following: 

a.  Management  approach 

b.  An  overview  of  the  requirements  analysis  approach 

c.  An  overview  of  the  approach  to  preliminary  and  detailed  design 

d.  Other  topics  to  be  determined  by  the  Offeror 

The  briefing  to  the  Government  shall  be  presented  after  delivery  to  the  Government  of  the 
exercise  products  described  in  points  a  through  d  in  3.1.  The  briefing  shall  not  include  any  discussion 
of  further  work  which  the  offeror  may  have  completed  following  submission  of  the  exercise  products. 
All  participants  in  the  exercise  shall  be  present  at  the  briefing  to  respond  to  government  questions;  all 
offeror  responses  to  these  government  questions,  together  with  the  briefing  presentation  material  and 
the  exercise  products,  are  considered  part  of  the  offeror's  proposal  and  subject  to  evaluation  by  the 
Government. 

4.0  SCOPE  OF  THE  SEE 

The  Government  will  not  evaluate  the  following  items: 

a.  Additional  work  accomplished  on  the  SEE  after  the  initial  four-week  period 

b.  Level  of  staffing 

c.  Measures  of  productivity 
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APPENDIX  B 


RFP  SECTION  M,  EVALUATION  CRITERIA  (Example) 


(This  is  only  gg  example!  This  section  must  he.  tailored.  fOL  each  source  selection  gad  included  in 

EEL  Section  M.  Evaluation  Criteria.  This,  material,  identifies  the  basis,  on  Which  <rn  offeror's  SEE 

products  will  be  judged  bv  the.  GQV£inm£llLl 

Item:  Software  Engineering  Exercise 

Offerors  are  evaluated  on  their  familiarity  with  the  selected  software  development  methods  and  on 
their  capability  to  utilize  Ada.  Offerors  are  also  evaluated  on  their  corporate  Ada  and  software 
engineering  expertise:  their  requirements  analysis  and  design  approaches  and  interrelationships;  the 
robustness  and  cohesion  of  their  requirements  analysis  and  design  methods;  their  familiarity  and 
expertise  with  the  methods;  their  familiarity  with  the  tool  set  and  the  development  environment;  the 
robustness,  cohesion,  and  completeness  of  their  exercise  design;  their  ability  to  address  and  analyze 
real-time  requirements  and  issues;  the  clarity  and  communication  of  their  design,  including  the  use  of 
ADL  to  express  design;  and  their  compliance  with  the  exercise  specification  requirements  and  their 
SDP.  A  visit  to  each  offeror  will  be  scheduled  after  receipt  of  their  SEE  products;  there  will  be  no 
opportunity  to  revise  the  exercise  products.  The  visiting  government  team  will  be  assisted  by  personnel 
from  the  MITRE  Corporation. 
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APPENDIX  C 


DETAILED  INSTRUCTIONS  FOR  THE  OFFERORS  (Example) 


(This  in  onN  an  example!  This,  document  should,  be  completely  tailored  for  each  SEE  since  the 

instructions,  products,  and  scope  would  differ  significantly  from  this  example.) 

1.0  PURPOSE 

The  purpose  of  the  Software  Engineering  Exercise  (SEE)  is  to  permit  the  Government  to  evaluate 
an  actual  application  of  each  offeror’s  software  development  method  as  proposed.  The  SEE 
concentrates  on  the  offeror’s  approach  to  requirements  analysis  and  design  and  their  interrelationship. 
The  offeror’s  approach  to  implementation,  integration,  test,  quality  assurance,  configuration 
management,  staffing  level,  productivity  measures,  software  metrics  collection,  and  other  development 
activities,  explicitly  mentioned  in  the  following  paragraphs,  is  also  evaluated  by  the  Government  as  part 
of  the  SEE.  {See  note  in  Appendix  A.) 

2.0  INSTRUCTIONS 

The  offerors  will  provide  a  prototypical  example  of  their  proposed  software  development 
approach  as  applied  to  the  exercise  problem.  Attachment  1  [appendix  D  of  this  document].  Problem 
Specification,  presents  the  requirements  for  the  problem.  In  performing  the  exercise  problem,  the 
offerors  shall  comply  with  all  provisions  of  their  proposed  Software  Development  Plans  (SDPs).  The 
offerors  shall  make  use  of  proposed  development  tools  and  procedures;  deviations  shall  be  noted  by 
each  offeror. 

Participation  in  the  exercise  shall  be  limited  to  those  individuals  identified  in  the  offeror's 
proposal  as  part  of  the  development  team,  subcontractors  who  will  be  responsible  for  software 
development  shall  be  active  participants,  consultants  are  precluded  from  participating.  Each  offeror 
shall  deliver  to  the  Government  all  requested  materials  in  the  formats  described  no  later  than....  The 
Government  will  review  this  material  for  a  period  of  time  not  to  exceed  two  (2)  calendar  weeks. 
Following  completion  of  the  Government  review,  a  Government  team  will  conduct  an  on-site  visit  at  the 
offeror’s  facility,  at  which  time  the  offeror  shall  brief  the  team  on  their  approach  and  provide  responses 
to  Government  questions  (the  Government  will  coordinate  the  schedule  for  the  on-site  visit  with  the 
offeror  upon  receipt  of  the  offeror’s  exercise  results).  Preliminary  plans  are  for  the  Government  to 
conduct  die  on-site  visit  during  the  week  of....  Note  that  there  will  be  no  interaction  between  the 
offeror  and  the  Government  during  the  offeror’s  implementation  of  the  exercise.  Should  the  offeror 
have  any  questions  on  the  exercise,  the  offeror  is  instructed  to  identify  appropriate  assumptions,  to 
document  the  assumptions,  and  proceed  with  the  exercise  based  on  these  assumptions. 

The  Government  will  conduct  its  evaluation  of  the  offeror’s  delivered  materials  and  assess  the 
offeror’s  proposed  methods  using  as  a  primary  reference  the  SDP  submitted  with  the  proposal, 
particularly  the  software  standards  and  procedures  contained  in  the  SDP.  The  offeror  may  submit  with 
the  SEE  products  an  augmentation  to  the  SDP,  not  to  exceed  fifteen  (15)  pages,  which  provides  further 
concise,  technical,  and  explicit  details  regarding  the  offeror's  proposed  software  development  approach 
and  methods.  The  Government  will  consider  any  such  augmentation  as  part  of  the  offeror’s  proposal 
(but  not  included  in  page  limitations),  and  subject  to  Government  evaluation. 

The  Government  will  employ  automated  tools  to  conduct  its  evaluation  of  the  offeror’s  delivered 
materials;  therefore,  the  offeror  is  required  to  deliver  some  of  the  exercise  products  in  machine-readable 
format.  In  order  to  assess  the  compatibility  of  the  Government’s  tools  and  the  offeror’s  machine- 
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readable  products,  the  offeror  is  requested  to  deliver  to  the  Government  no  later  than . a 

demonstration  tape  containing  sample  files  of  the  offeror’s  products  (for  example,  Ada-based  design 
language  (ADL)  listings)  in  the  same  format  that  will  be  submitted  at  the  conclusion  of  the  exercise 
period.  The  Government  will  not  evaluate  the  contents  of  this  demonstration  tape,  but  will  merely  use  it 
to  study  and  resolve  any  compatibility  issues  that  may  develop  between  the  Government's  tools  and  the 
offeror’s  tape  output  The  sample  files  on  the  demonstration  tape  do  not  need  to  represent  actual 
products  of  the  exercise;  they  need  only  represent  general  products  of  the  offeror's  proposed  methods 
which  the  offeror  will  submit  for  evaluation  at  the  end  of  the  exercise  period. 

3.0  PRODUCTS  OF  THE  EXERCISE 

3.1  Delivery  to  the  Government 

At  the  conclusion  of  the  exercise  period,  the  offeror  shall  deliver  the  following  items  to  the 
Government  for  evaluation: 

a.  A  complete  software  architecture  for  the  sample  problem.  This  architecture  shall  contain  an 
identification  of  CSCIs  and  CSCs,  an  allocation  of  functions  to  these  levels,  a  preliminary 
specification  of  interfaces,  and  internal  interface  diagrams  depicting  control  and  data  flow. 

b.  [For  one  or  more  Government-specified  components  of  the  system ,]  all  requirements 
analysis  conclusions  and  documentation.  With  respect  to  the  selected  components,  the 
requirements  analysis  shall  represent  a  complete  utilization  of  the  tools  and  procedures 
proposed  by  the  offeror.  Offerors  shall  identify  any  deviations  from  these  tools  and 
procedures  and  associated  rationale  for  these  deviations  in  their  briefings  to  the  Government 

c.  [For  one  or  more  Government-specified  components  of  the  system ,]  all  preliminary  design 
documentation,  including  requirements  traceability,  ADL  listings,  and  graphics  products. 
With  respect  to  the  selected  components,  the  preliminary  design  documentation  shall 
represent  a  complete  utilization  of  the  tools  and  procedures  proposed  by  the  offeror. 

Offerors  shall  identify  any  deviations  from  these  tools  and  procedures  and  the  associated 
rationale  for  these  deviations  in  their  briefings  to  the  Government 

d .  [For  one  or  more  Government-specified  component  of  the  system ,]  all  detailed  design 
documentation,  including  requirements  traceability,  ADL  listings,  and  graphics  products. 
With  respect  to  the  selected  component(s),  the  detailed  design  documentation  shall  represent 
a  complete  utilization  of  the  the  tools  and  procedures  proposed  by  the  offeror.  Offerors  shall 
identify  any  deviations  from  these  tools  and  procedures  and  the  associated  rationale  for  these 
deviations  in  their  briefings  to  the  Government. 

All  textual  products  of  the  exercise,  including  requirements  analysis  conclusions  and 
documentation,  ADL  listings,  and  other  design  documentation  shall  be  delivered  to  the  Government 
both  in  hard  copy  form  and  in  machine-readable,  9-track  tape  (exception  will  be  made  for  materials 
which  the  offeror  does  not  propose  to  create  or  maintain  on  line  during  the  contract.)  In  particular, 
graphical  representations  shall  be  submitted  in  hard  copy  form.  The  tape  shall  be  in  9-track,  1600  bpi 
format  in  accordance  with  ANSI  X3.27-1978,  ASCII  labelled,  and  with  an  identified  record  size  and 
block  size;  the  block  size  shall  be  512  bytes.  For  readability,  all  tabs  should  be  expanded  to  spaces. 

The  offeror  shall  provide  ten  (10)  copies  of  all  hard  copy  products;  the  products  delivered  shall  be  clear, 
coherent,  legible,  and  prepared  in  sufficient  detail  for  effective  evaluation  (elaborate  documentation, 
expensive  binding,  detailed  ait  work,  or  other  embellishments  are  unnecessary).  The  offeror  shall 
include  with  these  products  indices  delineating  the  subject  and  contents  of  the  hard  copy  material 
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package  and  the  9-track  tape,  the  operating  system  command(s)  used  to  create  the  tape,  a  list  of  ADL 
compilation  units,  and  a  list  of  the  compilation  order  of  these  units. 

3.2  Briefing  to  the  Government 

In  addition  to  the  delivered  products  described  above,  offerors  shall  provide  a  briefing  to  the 
Government  during  the  on-site  visit  that  summarizes  their  experience  in  carrying  out  the  exercise  and 
describes  the  products  generated.  The  briefings  shall  not  exceed  three  (3)  hours  in  duration.  The  topics 
presented  shall  include  the  following: 

a.  Management  approach,  to  include: 

1 .  Introduction  of  team  members 

2 .  A  description  of  individual  roles  and  experience 

b.  An  overview  of  the  requirements  analysis  approach,  to  include: 

1 .  A  rationale  for  the  selection  of  the  software  components 

2 .  A  description  of  the  tools  and  procedures  employed 

3 .  Significant  requirements  issues  encountered  and  their  resolution 

4 .  A  discussion  of  deviations  from  the  proposed  approach,  and  associated  rationale 

5 .  Other  topics  to  be  determined  by  the  offeror 

c.  An  overview  of  the  approach  to  preliminary  and  detailed  design,  to  include: 

1 .  A  rationale  for  the  selection  of  the  software  components 

2.  A  description  of  the  tools  and  procedures  employed 

3.  Significant  design  issues  encountered,  alternatives  considered,  and  a  rationale  for  the 
decisions  made 

4.  A  discussion  of  deviations  from  the  proposed  approach,  and  associated  rationale 

5 .  Other  topics  to  be  determined  by  the  offeror 

d .  Other  topics  to  be  determined  by  the  offeror 

The  briefing  shall  not  include  any  discussion  of  further  work  which  the  offeror  may  have 

completed  following  the  submission  of  the  SEE  products  on . .  since  the  Government  will 

not  evaluate  this  additional  work:.  All  participants  in  the  exercise  shall  be  present  at  the  briefing  to 
respond  to  government  questions.  The  offeror  shall  provide  ten  (10)  paper  copies  of  the  briefing  slides 
and  accompanying  text  at  the  time  of  presentation:  a  transcript  of  the  questions  and  answers  will  be 
kept  All  offeror  responses  to  these  government  questions  (the  transcript)  together  with  the  briefing 
presentation  material  and  the  exercise  products  identified  in  a  through  d  in  3.1  shall  be  considered  part 
of  the  offeror’s  proposal  and  subject  to  evaluation  by  the  Government 
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APPENDIX  D 


PROBLEM  SPECIFICATION  (Example) 


(This  is  only,  m  example  of  the  manner,  in  which  a  problem  should,  be  specified.  A  unions, 
problem  should,  be  developed,  for  eaeh  SEE,  that,  incorporates  time,  features  and  objectives  discussed,  in 

sections  7  and  2.} 

1.0  SCOPE 

The  exercise  system  will  create  scenarios  under  user  direction  and  will  simulate  the . 

capability  in  real  time. 

2.0  APPLICABLE  DOCUMENT 

System  Specification,  section  ....  dated . 

3.0  REQUIREMENTS 

3.1  General  Description 

The  exercise  system  shall  maintain  and  display  information  in  tabular  form  in  real  time. 
Specifically,  the  exercise  system  shall  create  scenarios  under  user  direction  and  store  each  created 
scenario  in  a  separate  scenario  file.  It  shall  use  a  generated  scenario  to  run  the  simulation  in  real  time. 

The  system  shall  provide  the  capability  for  the  user  to .  The  design  for  the  exercise  system 

shall  be  modular  to  facilitate  changes  in  software  components  which  are  needed  to  accommodate  future 
changes  in  operational  requirements. 

3.2  Hardware 

The  system  will  generate  only  tabular  displays.  No  special  graphics  hardware  or  capabilities  shall 
be  used.  The  user  interface  shall  be  designed  to  operate  on  a  single  dumb  terminal  with  a  keyboard 
entry  device. 

3.3  Simulation  data 
3.3.1  Configuration 

The  configuration  to  be  simulated  shall  be  as  follows: 

a.  There  shall  be  one  command  center. 

b.  There  shall  be  seven  sensors. 

c.  There  shall  be  five  missile  launch  origin  locations  and  five  predicted  impact  (nuclear 
detonation)  locations. 

d .  Sensor  connectivity  shall  be  from  each  sensor  to  the  command  center. 

e.  The  system  shall  simulate  the  transmission  and  processing  delay  incurred  from  the  time  a 
sensor  transmits  a  message  until  the  message  has  been  processed  by  the  system  and  made 
ready  for  display.  The  processing  delay  parameter  shall  be  user  selectable  from  0-99 
seconds  and  shall  be  constant  during  a  given  simulation. 
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3.3.2 


Data 


Data  shall  consist  of. . and . 

3.4  Display  Formats 

Display  formats  shall  consist  of  menus  for  the  user  interface  and  tabular  displays. 

3.4.1  User  Interface 

The  user  interface  shall  be  menu  driven  and  user  friendly.  All  user  input  shall  be  validated  for 
proper  format  and  range  of  values.  The  user  shall  be  notified  of  any  entries  that  are  erroneous  or  that 
cannot  be  processed  for  any  other  reason.  Error  messages  shall  be  self-explanatory  and  shall  specify, 
to  the  extent  practical,  the  cause  and  location  of  the  error. 

General  user  capabilities  to  be  provided  shall  include  the  capability  to  start  and  stop  a  session,  the 
capability  to  terminate  and  exit  to  the  main  menu  upon  user  request,  the  capability  to  display  the 
directory,  the  capability  to  select  the  processing  delay  parameter,  and  the  capability  to  interface  with 
.  and  . 

All  user  input  shall  be  acknowledged  within  one  second  of  the  input.  For  data  entered  by  the 
user,  the  time  from  completion  of  entry  until  the  data  base  is  modified  to  reflect  the  update  shall  not 

exceed  two  seconds.  An  advisory  shall  be  provided  within . seconds  if  the  system  cannot  complete 

such  an  update.  At  a  minimum,  these  performance  requirements  shall  be  met  on  dedicated  processing 
equipment  and  with  at  least  20  stored  scenario  files,  consisting  on  the  average  of . events. 

3.4.3  Tabular  Displays 

The  system  shall  be  able  to  generate  three  displays  for  data:  a  Summary  Display,  a  Predicted 
Summary  Display,  and  a  Message  Display.  The  summary  displays  shall  present  the  information  as 
generated  by  a  selected  scenario,  summarized  from  the  start  of  the  scenario,  in  real  time,  and  in 
accordance  with  the  specified  processing  delay.  The  formats  for  the  Summary  Display  and  the 

Predicted  Summary  Display  shall  be  as  specified  in . The  Message  Display  shall  sequentially  list 

the  messages  received  by  the  command  center  as  received  in  real  time.  The  capability  to  display  the 
contents  of  at  least  the  five  most  recently  received  messages  in  the  scenario  shall  be  provided.  Display 
updates  shall  be  processed  and  reflect  a  scenario  event  within  one-half  second  of  the  activation  time  of 
the  event. 

3.5  Scenario  Generator 

The  Scenario  Generator  (SG)  shall  be  activated  and  deactivated  only  as  a  result  of  user  action. 

The  SG  shall  be  able  to  create,  delete,  edit,  and  save  files  containing  scenario  data.  Edit  capabilities  for 
a  selected  scenario  file  shall  include  changing  the  contents  of  events  in  the  scenario  file.  The  capability 
shall  be  provided  to  save  a  scenario  and  any  changes  to  it  as  a  new  file  or  as  the  current  file.  Each  event 
in  a  scenario  shall  have  a  unique  activation  time  to  the  nearest  tenth  of  a  second,  where  the  activation 
time  represents  the  time  the  reporting  sensor  transmits  the  missile  warning  message.  The  user  shall  be 
precluded  from  entering  multiple  events  into  a  scenario  with  the  same  activation  time.  The  user  shall  be 
able  to  query  an  individual  scenario  file  to  search  for  events  based  on  reporting  sensor  and/or  time  of 
event  activation.  The  design  for  the  system  shall  be  flexible  to  allow  the  capability  to  perform  this  query 
across  all  scenario  files.  The  SG  shall  accept  input  from  the  keyboard  to  perform  the  above  functions. 
There  shall  be  a  default  scenario  file  consisting  of  a  total  of . individual  events  and  their  associated 
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times  of  activation  covering  a . minute  scenario  period.  The  SG  shall  support  a  total  of  at  least . 

events  contained  in  one  or  more  scenarios. 

3.6  Simulation 

The  simulation  shall  provide  the  user  with  the  capability  to  select  and  run  a  scenario  contained  in  a 
scenario  file.  It  shall  run  this  scenario  in  real  time,  generating  the  Summary  Display,  the  Predicted 
Summary  Display,  or  the  Message  Display  as  specified  by  the  user.  The  simulation  shall  be  activated  or 
deactivated  only  upon  user  request.  Capabilities  shall  be  provided  for  the  user  to  select  the  processing 
delay  parameter,  to  suspend  the  simulation,  to  resume  the  simulation,  to  fast  forward  the  simulation 
(where  fast  forward  means  the  run  time  between  event  activations  is  reduced  by  two),  and  to  stop  the 
fast  forward  capability  and  return  to  the  normal  run  time  between  event  activations.  The  user  shall  also 
have  the  capability  to  select  among  the  three  displays,  and  to  move  to  other  displays  while  the 
simulation  is  running. 

3.7  Simultaneous  Generation  and  Simulation 

The  system  shall  provide  the  capability  for  the  user  to  run  the  simulator  and  SG  simultaneously, 
either  on  the  same  or  different  scenario  files,  while  still  meeting  the  performance  requirements  specified 
herein.  Formats  for  the  displays  when  both  are  running  simultaneously  will  be  defined  by  the 
contractor  as  part  of  the  design  effort 

When  both  the  SG  and  simulator  are  processing  the  same  scenario,  the  simulator  displays  shall 
reflect  a  modification  to  an  event  in  the  scenario  only  if  the  event  has  not  yet  been  processed  by  the 
simulator,  otherwise,  the  simulator  displays  shall  not  reflect  the  changes. 
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